Skip to content

Add text selection system with frame-level cell composition - #980

Closed
chiga0 wants to merge 3 commits into
vadimdemedes:masterfrom
chiga0:feat/text-selection
Closed

Add text selection system with frame-level cell composition#980
chiga0 wants to merge 3 commits into
vadimdemedes:masterfrom
chiga0:feat/text-selection

Conversation

@chiga0

@chiga0 chiga0 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Problem

Background: why full-screen TUIs need virtual scrolling

Long-running interactive applications — coding assistants, chat clients, log viewers — accumulate content that far exceeds the terminal height. A single Qwen Code session can produce thousands of lines of conversation, tool output, and code blocks. Rendering all of this into the terminal's scrollback buffer creates two problems:

  • Performance. Every re-render (on each keystroke, streaming token, or state change) requires Ink to erase and repaint the entire output region. With hundreds of lines in the scrollback, this causes visible flicker and high CPU usage. The terminal must process thousands of ANSI erase/write sequences per frame.
  • Layout control. The application cannot anchor UI elements (input prompt, status bar, sticky panels) to fixed screen positions when the output region grows unboundedly into the scrollback.

Virtual scrolling solves both: the application enters alternate-screen mode (?1049h) to get a fixed-size viewport, then only renders the visible slice of content. As the user scrolls, the viewport window moves over the full dataset and the render output is recycled — much like a virtualized list in a GUI framework. The terminal sees a constant-height frame, eliminating flicker and enabling precise layout.

The selection problem

Alternate-screen mode and virtual scrolling together break text selection, which is a basic user expectation:

  1. The scrollback buffer is empty. The alternate screen (?1049h) is a separate, fixed-size buffer with no scrollback history. Terminal emulators' native selection (click-drag, double-click word select, etc.) operates on the scrollback buffer. In alternate-screen mode there is nothing to select from — the content exists only as the current frame, which the terminal treats as ephemeral (by design — think vim, htop, less).

  2. Virtual scrolling means the full content is never on screen. Even if a terminal emulator supports selection within the alternate screen, it can only select what's currently rendered in the viewport. Content that has been scrolled off is not in any terminal buffer — it exists only in the application's data model. Selecting across a range that spans multiple viewport-fuls of content is impossible through terminal-native mechanisms.

  3. Ink's render pipeline is opaque to the application. Ink composites the yoga layout tree into a string of ANSI sequences and writes it to stdout. The application has no structured access to what's actually on screen — which characters are at which cell coordinates, which are selectable, how lines wrap and flow. Without this information, the application cannot implement its own selection that maps mouse coordinates to logical text content.

The result: users of full-screen Ink TUIs cannot select and copy text — code blocks, error messages, conversation content. For a coding assistant where copying generated code is a core workflow, this is a critical usability gap.

Solution

This PR adds a frame-level cell composition system that gives applications structured, per-cell access to the rendered frame, enabling mouse-driven text selection within Ink's rendering pipeline.

The key insight: Ink already walks the yoga tree and writes styled characters into an Output grid during rendering. This PR extends that grid with per-cell metadata (selectability, reading-flow grouping, line-break semantics) and exposes it through a FrameController bridge:

Application                          Ink Renderer
    |                                     |
    |-- setSelection({sx,sy,ex,ey}) ---->|  (schedules repaint)
    |                                     |
    |<---- publishFrame(cells[][]) -------|  (after each render)
    |                                     |
    |-- getFrame() / subscribe() -------->|  (read latest frame)

The application handles mouse events and selection logic; Ink provides the composited frame data and applies the selection highlight during serialization. Selection is a pure render-time concern — it never mutates committed frame state.

Public API

getFrameController(stdout): FrameController | undefined

Access the frame controller for a render instance (via the existing instances map).

FrameController

Method Description
getFrame(): ReadonlyFrame | null Latest composited frame (cells + boundaries)
getSelection(): ScreenSelection | null Current selection coordinates
setSelection(sel | null): void Update selection, schedule repaint if changed
subscribe(listener): () => void Observe frame publications

FrameCell

Each cell in the composited grid carries:

  • type, value, fullWidth, styles — existing styled character data
  • selectable: boolean — whether this cell participates in selection
  • flowId: number | null — logical reading flow grouping

New <Text> props

Prop Type Default Description
selectable boolean true Whether the text participates in selection
selectionFlow string Group texts into a logical reading flow
selectionBreakAfter 'soft' | 'hard' How the line break after this text behaves during copy
selectionJoiner string '' Joiner inserted when reconstructing across this break

Internal changes

  • Output.get(selection?) — returns {output, height, cells, boundaries}. Cells carry selectable/flowId metadata. When a selection is provided, selected cells get a highlight background applied before serialization.
  • renderNodeToOutput — threads flowIds/nextFlowId maps through the tree. Uses wrapTextWithMetadata instead of plain wrapText to track soft/hard line boundaries and per-row selectability.
  • wrapTextWithMetadata — wraps text and returns {text, boundaries: TextBoundary[], selectableRows: boolean[]} for semantic copy reconstruction.
  • renderer — accepts an optional selection parameter, creates flow tracking state, returns cells/boundaries alongside the string output.
  • Ink class — creates a FrameController in the constructor, publishes frames after each render, reads selection state for highlight.

Testing

All 86 existing component tests pass. The feature is additive — no existing behavior changes when selection is not used.

This feature has been running in production in Qwen Code's TUI (via a patch on ink 7.0.3) for several months, supporting text selection in a full-screen alternate-screen TUI with virtual scrolling.

秦奇 and others added 3 commits July 27, 2026 16:50
Introduce a bidirectional bridge between the application and the
renderer for terminal text selection. The renderer now composites
each frame into a cell grid (FrameCell[][]) with per-cell selectability
and semantic flow metadata, and can apply a selection highlight before
serialization.

New public API:
- getFrameController(stdout) — access the frame controller for a
  render instance
- FrameController.setSelection(sel) — schedule a repaint with the
  given selection highlighted
- FrameController.getFrame() / subscribe() — read or observe the
  latest composited frame

New <Text> props:
- selectable (default true) — whether the text participates in
  selection
- selectionFlow — group texts into a logical reading flow
- selectionBreakAfter / selectionJoiner — control how line breaks
  are reconstructed during copy

Internal changes:
- Output.get() returns cells and boundaries alongside the string
- renderNodeToOutput threads flow IDs and uses wrapTextWithMetadata
  for semantic line-break tracking
- wrapTextWithMetadata wraps text and returns boundary/selectability
  metadata per row
Resolve the 35 lint errors blocking CI, all behavior-preserving:

- Replace `null` types and runtime literals with `undefined` to match
  the codebase convention (@typescript-eslint/no-restricted-types).
- Use `Array<T>` / `readonly T[]` per @typescript-eslint/array-type.
- Use Unicode escapes with uppercase hex for the selection background
  sequence (unicorn/escape-case, unicorn/no-hex-escape).
- Rename SELECTION_BG → selectionBackground (naming-convention).
- Replace Array#reduce with an explicit loop (unicorn/no-array-reduce).
- Move the public frameController field before the private fields
  (member-ordering) and drop an unnecessary type assertion.

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
Add test/selection.tsx exercising the public selection API:

- getFrameController lifecycle (undefined for unknown stdout, present
  after render)
- getFrame exposes composited cells with correct dimensions/values
- default vs selectable={false} cell flags
- setSelection highlights the selected region, clearing removes it, and
  identical selections are deduplicated (no extra repaint)
- subscribe receives published frames
- selectionFlow groups flowIds; distinct nodes get distinct flowIds
- selectionBreakAfter="hard" records a hard boundary

Generated with AI

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
@chiga0

chiga0 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

Pushed two follow-ups to get this review-ready:

  • CI is green now. Resolved the xo lint errors that were failing the build — all behavior-preserving (nullundefined per the codebase convention, Array<T>/readonly T[], Unicode escapes, a const rename, reduce → loop, member ordering).
  • Added test coverage in test/selection.tsx (12 tests) for the public API: getFrameController lifecycle, getFrame() composited cells, selectable flags, setSelection highlight/clear/dedup, subscribe, selectionFlow grouping, and selectionBreakAfter boundaries.

@sindresorhus @vadimdemedes whenever you have time to take a look, thank you!

@sindresorhus

Copy link
Copy Markdown
Collaborator

I think the problem is worth solving, but I would not merge this PR as-is.

The raw frame/controller bridge makes sense for apps that own alternate-screen input and need to implement selection themselves. Qwen Code is a good example. In normal terminal mode, native terminal selection is still simpler, so this should be scoped to the app-owned viewport use case.

The main issues are:

  1. Metadata is lost when nested Text nodes are squashed. For example, selectable={false} on an inner Text is published as selectable, which is a correctness issue for an existing Ink composition pattern.
  2. Reverse selections are not normalized, so right-to-left selections can silently select nothing.
  3. Subscribers are called synchronously while rendering. A subscriber that changes selection can re-enter rendering and potentially write frames out of order.
  4. Frame and semantic data are generated on every render, even when no consumer uses selection. This should be opt-in or benchmarked carefuly.
  5. The public API is larger than it needs to be. It exposes publishing and internal style data, while the supposedly read-only frame is not deeply immutable.
  6. The tests do not cover nested text, reverse selections, wide characters, nonselectable highlighting, or subscriber reentrancy.

I would split this into two PRs. First, add a small opt-in read-only frame controller for the visible alternate-screen viewport. Keep publishing internal, define the coordinate semantics, and expose only the data a consumer needs. Then add flows, boundaries, and semantic Text props in a follow-up once the copy behavior is proven by a real consumer and fixture suite.

So: yes to the capability, but no to this PR in its current shape. I would simplify and narrow the first step before adding more selection semantics.

@chiga0

chiga0 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of #984 — the first part of the split suggested in review (small opt-in read-only frame controller for the alternate-screen viewport). This branch stays around as the basis for the follow-up PR covering flows, boundaries, and semantic Text props once the copy behavior is proven by a real consumer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants